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Abstract 


On May 11-13, 1994, JPL's Center for Space Microelectronics Technology (CSMT) 
hosted a neural network workshop entitled, "A Decade of Neural Networks: Practical 
Applications and Prospects," sponsored by DoD and NASA. The past ten years of 
renewed activity in neural network research has brought the technology to a crossroads 
regarding the overall scope of its future practical applicability. The purpose of the 
workshop was to bring together the sponsoring agencies, active researchers, and the user 
community to formulate a vision for the next decade of neural network research and 
development prospects, with emphasis on practical applications. Of the 93 participants, 
roughly 15% were from government agencies, 30% were from industry, 20% were from 
Universities, and 35% were from Federally Funded Research and Development Centers 
(FFRDCs). 

The workshop began with a keynote address by Dr. Dwight Duston, Director of the 
Ballistic Missile Defense Organization/Innovative Science and Technology Office 
(BMDO/IST), who gave an overview of past, present, and future government and industry 
investment strategies for neural networks, with an emphasis on the need for a strong 
commercialization effort for this technology. A series of invited talks followed, which 
focused on a variety of applications both in control and signal processing. Topics such as 
fault diagnosis (vehicle engine health monitoring for automotives, space launch vehicle 
anomaly detection), pattern recognition (document analysis, automatic target recognition, 
artificial vision, face recognition), multiparameter optimization (unsteady aerodynamic 
control, resource allocation, path planning), image compression (video-conferencing, 
interactive education, home entertainment), and telecommunications (broadband access 
control, equalization, fraud detection, software reliability prediction) were covered in the 
presentations. A separate workshop proceedings, JPL Publication 94-10, includes 
abstracts and excerpts of presentation materials submitted by the invited speakers. 

Following the talks, parallel working groups were formed. The working groups discussed 
the current state of the art; the competitive advantages of neural networks with respect to 
alternative methods; the high impact, high payoff directions to commercializing neural 
network applications; addressing issues such as technology insertion, technology 
acceptance, and enabling technologies; and finally prospects of industry-academia- 
govemment collaborative research and development This report summarizes the 
discussions and recommendations of the working groups. 
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Section I. 


Introduction 


For over a decade, JPL has actively pursued neural network research on theory, 
architectures, algorithms, applications, and electronic as well as optical hardware 
implementations. The work at JPL has been sponsored primarily by NASA and several 
DoD agencies. As a Federally Funded Research and Development Center (FFRDC), JPL’s 
role in this technology area has been complementary to U.S. industries and universities. 
JPL has focussed on technical issues, where JPL can offer certain "technology advances, 
unique facilities, and special expertise" that complement the ongoing work at universites 
and US industries. As part of the ongoing technology assessment for our sponsors, JPL 
hosted this workshop. The overall purpose of the workshop was to bring together the 
sponsoring agencies, active researchers, and the user community, to formulate a vision for 
the next decade of neural network research and development prospects, with emphasis on 
practical applications. 

In order to focus the workshop on the practical aspects of neural networks, a series of 
invited talks emphasizing real world applications both in control and signal processing was 
presented. Topics such as fault diagnosis (vehicle engine health monitoring for 
automotives, space launch vehicle anomaly detection), pattern recognition (document 
analysis, automatic target recognition, artificial vision, face recognition), multiparameter 
optimization (unsteady aerodynamic control, resource allocation, path planning), image 
compression (video-conferencing, interactive education, home entertainment), and 
telecommunications (broadband access control, equalization, fraud detection, software 
reliability prediction) were covered in the presentations. A separate workshop 
proceedings, JPL Publication 94-10, includes abstracts and excerpts of presentation 
materials submitted by the invited speakers. 

Following the invited presentations, three splinter groups were formed for parallel 
breakout sessions, to formulate recommendations for future neural network research and 
development. The remainder of this report summarizes their discussions and 
recommendations. 

Section II. Working Group Summaries 

In each of the three parallel splinter groups, emphasis was placed on applications of 
practical significance. Two of the three groups focused on the application areas of control 
and signal/image processing, respectively, whereas the third group addressed all 
application areas in general. The groups had representation from large as well as small 
industries, FFRDCs, government agencies, and academia. The group leaders represented 
small industry, large industry and government. Drs. Kenneth Marko and Lee Feldkamp of 
Ford Motor Company led the group on control applications. Demetrios Sapounas of the 
Naval Surface Warfare Center led the discussion on signal and image processing 
applications, and Dr. Azad Madni of Intelligent Systems Technology, Inc., led the third 
group on control, signal processing, and all other application areas in general. 
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During the four hour time period, the splinter groups addressed four topical questions in 
developing their recommendations. The questions were: 

• What can neural networks do today that is unique or substantially better, compared to 
alternative techniques? What are the advantages of neural networks? 

• What are some of the emerging high impact/high demand applications of neural 
networks and the technology(ies) needed to support them? 

• What are the predominant barriers to neural network technology insertion/acceptance? 
How can they be overcome ? 

• What roles should government, FFRDCs, industry and academia play in realizing the 
future of artificial neural networks? 

There was significant overlap in the conclusions of the three groups. The following is a 
summary of the discussions, prepared primarily from the remarks of the splinter group 
leaders, highlighting relevant discussion points in each of the four topical areas. 

1) Neural Networks Today 

What can neural networks do today that is unique or substantially better, compared to 
alternative techniques? What are the advantages of neural networks? 

Although the splinter groups were charged with the primary task of identifying some of 
the real "success stories" for Artificial Neural Networks (ANNs), almost every group 
directly or indirectly addressed the important question: "what neural networks cannot do 
and why?" Clearly, ANNs are playing an important role in the way people think about 
problems today. Many requests have surfaced to study problems using neural networks 
that were regarded as too difficult to tackle before ANN paradigms became widely 
known. However, it should be realized that ANNs represent only certain computational 
paradigms with their own specific strengths and weaknesses. ANNs by themselves are not 
a panacea for solving all problems. In fact, there are currently numerous efforts underway 
to combine and complement ANNs’ capabilities with one or more other computational 
techniques such as fuzzy logic, Al/expert systems, and genetic algorithms. In defense of 
ANNs, however, there are many occasions when unrealistic expectations are placed on 
neural networks and complex (or sometimes even unsolvable) problems are improperly 
posed to them. Further, the ANN methodology is not yet developed to the degree that 
non-specialists can successfully apply the techniques to problems of any complexity they 
desire. As a result, there is sometimes a mismatch between the researcher or engineer 
capability and the complexity of the problem, leading to an abortive attempt to solve a 
problem, and subsequent dismissal of ANN technology. 

From the examples presented by the discussion group members, however, it was clear that 
when properly applied, neural networks have indeed delivered results significantly better 
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than other competing techniques in numerous cases. The following is a cross section of 
the cases brought out by the group members with first-hand experience, where ANNs 
have clearly left a mark. 

After the emergence of neural networks, adaptive learning and trainable classifiers based 
on neural networks have become a rather routine part of many researchers' mathematical 
toolkits. Neural networks' unique capabilities in handling non-linear relationships and 
capturing ill-defined transformations from examples, where a complete model, data, or 
a priori knowledge is not available have led to a wide variety of applications. For instance, 
in the area of time series forecasting, ANNs are being put to the test on perhaps one of the 
most ill-defined temporal problems: predictions in financial markets. The London Office 
of the Union Bank of Switzerland (USB) is actively assessing the effectiveness of neural 
networks on predictions in the currency and mutual funds trading markets. Another 
growing application area for ANNs is pattern recognition for medical visualization aids 
and image processing. The commercial product, Papnet, developed by Neural Medical 
Systems, New York, performs PAP smear analysis (cancer screening) and is based on a 
neural implementation. Currently, the Martin Marietta Corporation in collaboration with 
Rose Health Care Systems is putting a mammogram analysis system in service for on-line 
screening of cancerous lesions. The system utilizes optical Fourier bandpassing and 
wavelet filtering to preprocess the image followed by a neural network for analysis and ill- 
defined pattern identification with final confirmation performed via optical correlation. 
Alacron Inc. is utilizing neural networks to perform classification on clinical neuro- 
physiological data, such as electro-encephalograms (EEGs) and Evoked Potentials (EPs), 
which show consistently accurate classification results even in the presence of noisy and 
corrupted recordings. This automated pattern identification and analysis system is being 
developed for real time acute care monitoring in the ICU and operating room setting. 

In addition to these examples, there are several emerging ANN implementations and 
applications, such as optical character recognition (developed by AT&T for the US postal 
service), environmental controllers, image compression techniques, automotive engine 
health monitoring, and system modeling. Although in many such cases, the technical 
advantages of neural network methodology are clear, neural networks have not yet been 
accepted as routine procedures. 

If neural networks clearly offer computational benefits that could be translated into cost 
reduction, speed enhancement, and/or system (hardware/software) simplicity, then what 
has stopped them from being widely accepted and inserted in large numbers of 
applications? A majority of the participants acknowledged that the field of neural 
networks has not only had a long history, but it has also received significant research 
dollars, federal as well as commercial, particularly in recent years. Therefore, for the field 
of neural networks to remain alive and vibrant, it must enter into real-world applications in 
a substantial way in the near future. There are clearly numerous application possibilities 
that could, in principle, become economically viable at this time. However, unless the 
early applications result in a broad market penetration through a notably high demand or 
high impact, the field of neural networks may once again drift and possibly dissolve into 


3 



some other all-pervading computational paradigm. This discussion, therefore, led the 
groups in a natural way to the next posed question: 

2) Future Applications and Relevant Technologies 

What are some of the emerging high impact! high demand applications of neural networks 
and the technology(ies) needed to support them? 

This part of the discussion generated a truly overwhelming list of applications. The 
applications could be divided into two broad classes. The first set of applications have 
emerged from several narrowly defined but quite complicated requirements of DoD, 
NASA and other agencies, which are driving certain application-specific neural network 
development efforts (e.g. various forms of real-time automatic target recognition, 
guidance and control of an interceptor, autonomy in planetary rovers, etc.). Although 
results of such efforts would be complex high performance neural network systems, 
immediate demand for such systems would perhaps be low, until the subsequent 
development of commercial counterparts of these sophisticated systems. The second set 
of applications, primarily characterized by their potential high demand, would arise from 
the commercial markets. Four broad application areas of neural networks that were 
viewed as potentially high impact or high demand candidates by a majority of the 
participants are described below. 

A) Future Applications: 

Transportation 

One of the identified areas where neural networks are expected to have high demand and a 
significant (performance) impact in the near future is transportation. The applications 
would range from vehicle health management for (automotive) engines leading to lower 
emission and higher efficiency transportation to intelligent vehicles and better traffic 
management In the case of power train diagnostics and control, for example, VLSI 
implemented compact neural network circuits could become an integral part of "under- 
the-hood" engine computers. It was recognized that this area truly represents a high 
demand (the sheer number of vehicles) and could be substantially impacted by low cost, 
easily deployable, neural network circuits. Furthermore, neural net classification 
techniques may also be incorporated into drunk, fatigue, and/or high G monitors. 
Automated route planning, both centralized and distributed in vehicles, has a potential to 
become an integral part of the Intelligent Vehicle Highway System (IVHS) and could 
greatly benefit from neurally inspired algorithms [1]. 

Health Care 

The vast medical field has a lot to gain from the applications of neural networks to a 
variety of ill-defined pattern recognition, classification, multiparameter data interpretation 
tasks, and control problems. Most of these tasks are currently performed by humans. 
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usually visually, with a significant subjectivity and uncertainty in the outcome. In the near 
future, neural network-inspired techniques are expected to appear in a variety of ill- 
defined classification, trend analysis, and overall medical data/image interpretation tasks to 
assist a medical practitioner with better analytical tools (i.e., medical aids). Clearly, due to 
the legal ramifications of implementing any new techniques in the medical world (and the 
difficulty of getting Federal Drug Administration (FDA) approval for new equipment, 
etc.), commercial development of such applications must follow a slow and arduous path. 
Perhaps the future revamping of the medical system with an emphasis on cost savings may 
allow better reception for introduction of such new technologies. Applications such as 
monitor automation (e.g., EKG or multi-sensor on-line) and medical life support control 
will have to await changes in the medical/legal system. Physiological signal-driven 
artificial limbs or prostheses may also incorporate neural algorithms, hardware, or simply 
physiological understanding derived from neural net research. 

Industrial robotics, manufacturing, and process control: 

Industrial robotics could be heavily impacted by neural networks in the time frame of the 
next ten years. The specific areas range from automated quality inspection on assembly 
lines to fault-tolerant, adaptive sequencing/scheduling and resource allocation operations. 
In addition, on-line process control in chemical plants, based on multi-sensor multi-modal 
input parameters, is an ideal problem for neural network solutions. 

Human-machine interface: 

The next high demand application area for neural networks identified by the workshop is 
in the rapidly evolving human-machine interface field. In particular, hand-wntten 
character recognition, large vocabulary speech recognition, and speech synthesis are the 
applications attracting considerable attention today. Also, several participants believed 
that combinations of fuzzy logic and neural networks are going to find wide-spread 
applications in the day-to-day situations of home security, home health monitoring/energy 
management, and home appliance monitoring/maintenance (i.e., "smart house"). 

B) Relevant Technologies: 

Although the field of neural networks is not new and a variety of useful, real-world 
applications with clear benefits is already in sight, our understanding of the variety of 
algorithms used by real biological neural networks and their architectural details has only 
scratched the surface. Clearly, more research is required to develop a better 
understanding of the richness and fidelity of biological nervous systems in order to exploit 
the full potential of ANN. Better theoretical underpinnings are needed, especially in the 
areas of biological modeling and learning techniques (e.g., bifurcating or spiking 
networks). Further, the scalability of many of today's popular ANN algorithms for 
supervised and unsupervised learning needs to be explored and established. Other areas 
that should receive researchers’ attention include better data representation in large 
networks and inexpensive scalable and accessible hardware. Improved analog- to-digital 


5 



interfaces and tight coupling of sensors and processors were also cited as important 
supporting technologies for future high impact/high demand applications. 

The overall goal of artificial neural networks is to learn from nature's successes and to mimic 
them in order to accomplish intelligent tasks not easily performed by conventional (sequential) 
computing techniques. The continuing evolution of life on the planet has experimented with a 
large variety of sensing devices and information processing architectures in biological 
organisms, covering an enormous spectrum of capabilities and varying degrees of intelligence. 
An important architectural aspect of the biological "processors" is their full and direct 
integration with biological sensors u tilizin g highly parallel, direct, and therefore quite efficient 
interfaces. The variety of biological computing architectures, sensors, and the communication 
links between them is truly amazing. To dramatically enhance the overall effectiveness of 
ANNs, therefore, sensor-processor integration must receive serious attention. An example of 
such tight coupling between sensors and processors, discussed at the workshop and described 
below, involves the integration of low-power analog VLSI ANN chips in a 3-dimensional cube 
architecture, which receive parallel input directly from the back of a focal plane sensor array. 

Sensor-processor integration for smart vision: 

A technology opportunity that lends itself to artificial-vision-related applications utilizing 
tight coupling between sensors and processors is the 3-D Artificial Neural Network 
(3D ANN). The 3DANN consists of a stack of thinned VLSI neural network VLSI chips 
directly mated to an imager array [2]. Offering high density and massively parallel "focal 
plane" processing, this smart IC "sugarcube" could be used in a variety of applications, 
such as a smart "eyeball" for industrial robotics, autonomous space exploration, and 
automatic target recognition for BMDO's interceptor application. The cube offers 
tremendous speed potential. (For example, nominally a cube with a 128 x 128 pixel array 
as a front end will achieve over 10^ interconnects/sec processing speed, whereas a cube 
with 1024 x 1024 pixel array will surpass 10^ interconnects/sec). However, there are 
major challenges in learning to train, control, and develop algorithms for, and 
communicate with, such a network that need to be addressed in future research to be able 
to put the unprecedented computing power of such a cube to use in complex applications, 
going beyond current image processing functions. 

3) Overcoming Barriers to Technology Insertion/Acceptance 

This topic generated the most compelling discussions in terms of what needs to be done 
differendy in the future to facilitate the insertion of neural network ideas and technology 
into the mainstream. In addition to the legal ramifications discussed above, which must be 
addressed through proper channels, and the general limitation on research funds in the 
current fiscally lean times, three major areas surfaced as barriers to technology acceptance: 

1) Perception problems 

2) Lack of technical understanding on the part of the prospective user community 

3) Lack of coordination 
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These barriers as well as recommended solutions for them are presented below. 

Perception 

Neural networks have suffered from a number of perception problems which can be 
attributed to a variety of causes, most notably poor marketing on the part of researchers. 
The name "neural" itself without proper understanding of the subject has been a major 
issue, with reactions ranging from fear due to certain religious beliefs to the perception of 
science fiction. Due to the excessive hype ANNs encountered several years ago, there 
have been inflated performance expectations, eventually leading to disappointment with 
the technology and its cost effectiveness. 

Most of these perception-related problems could be addressed with a good marketing 
strategy and improved advertising. Compelling demonstrations and user involvement early 
in the development cycle will also dispel some of these false impressions. Education and 
training discussed in more detail below will also improve the appeal of neural networks. 
Finally, incremental acceptance as ANN products slowly filter into systems will gradually 
change these perceptions. 

Education 

The lack of technical understanding may be the biggest barrier to technology insertion, but 
it is also the easiest to address with the proper allocation of funds. Explicit knowledge 
about neural systems is still limited, resulting in a "black box” syndrome (a lack of 
understanding of the decision-making properties of an artificial neural network) and 
wariness to employ such networks. With only a few rigorous courses in neural networks 
in the engineering curricula, there is a reluctance on the part of engineers to utilize neural 
networks. Specifically, this reluctance is reinforced by their other training which 
emphasizes a model-based approach to problem solving, contrary to the model-free ANN. 
These problems can be clearly addressed by appropriate education at all levels. By 
creating an interdisciplinary core curriculum which exposes students to neural network 
ideas, both engineers and non-engineers (e.g. future business managers) will be more 
receptive to novel approaches for problem solving. In addition, a theory akin to classical 
circuit theory and associated textbooks need to be developed for use in the engineering 
curricula. 

Several of the workshop participants argued that it is important to develop guidelines and 
criteria for "ANN architecture optimization" and "training-set selection" for a given class 
of problems for prospective users of ANNs. Finally, there is a lack of system development 
knowledge and domain understanding, as well as confusion between "adaptation" and 
learning, that needs to be addressed. Teaming with system engineers and domain experts 
as well as sponsoring tutorial presentations and educating management will smooth the 
way to technology acceptance. 
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Coordination 


There are several other important barriers to technology insertion that can be loosely 
associated with problems of coordination. One is a competition with artificial intelligence 
(AI) and classical techniques, which can be easily overcome by fostering complementarity 
with these techniques. Neural networks with their unique set of strengths and weaknesses 
need to be utilized in conjunction with other methods and there is no need for competition, 
especially with insufficient knowledge of either of the technologies. However, it is 
important to provide relevant benchmarking between differing techniques so that the 
technology can be accepted and understood within the existing knowledge framework. 
For example, the wrong attributes are often compared, such as the apparent inaccessibility 
of the internal representation of the problem/situation generated by an ANN with an easily 
accessible (often incomplete, distorted, or totally inappropriate) plant model while using 
other rule-based techniques. This can easily lead to misunderstanding and missed 
opportunities. Finally, a direct connection between research investment and revenue 
generation needs to be established to further encourage applied funding opportunities. 
The workshop recommended both introducing ANN technology into existing products in 
an incremental add-on style (e.g., on-line engine health management in automotive 
industry) as well as bringing totally new capabilities into innovative high demand markets 
(medical image and data interpretation). 

4) Roles of Government, FFRDCs, Industry and Universities 

What roles should government, FFRDCs, industry and universities play in realizing the 
future of artificial neural networks? 

The participants acknowledged the increasing and fruitful interactions among the 
FFRDCs, industry, and academia over the last few years, encouraged by new and old 
mechanisms for formal relationships, such as the Small Business Innovative Research 
(SBIR), Small Business Technology Transfer (STTR), Technology Reinvestment Project 
(TRP), and Advanced Technology Program (ATP) grants and agreements. However, the 
groups felt that there is always room for improvement in coordination, primarily to avoid 
expensive duplication of efforts. In addition to the general roles for each of the parties, 
the groups also came up with some specific suggestions. For example, it was expressed 
that the government's attempt to catalyze commercial applications of emerging 
technologies through new programs such as TRP and ATP is excellent It strongly 
encourages industry to make serious investments in high payoff technologies. However, 
the government needs to continue funding high risk research and development areas in 
ANNs that will not be undertaken by industry. Overall, the government's role would be in: 

• Infrastructure development 

Guide consortia creation and operation 

• Intelligent funding 

Select high payoff/high risk efforts that would not be undertaken by industry 
Fund needed support technologies (hardware, theoretical work) 

Fund both university research and curriculum development 
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FFRDCs in their unique non-profit role should support government in technology 
assessment, policy making, and program development. Their role in research and 
development should be restricted to only such activities that are not being undertaken by 
industry or academia. Furthermore, they must spend significantly more effort in transfer 
of their technology to industry in a timely fashion. 

Industry, on the other hand, must lead ANN technology commercialization and 
proliferation, as they would be the immediate and first beneficiaries. Large industries 
should actively work on opportunities for collaboration with small businesses and 
universities. Their role could also include: 

• Creation of forums for information exchange (to expose small businesses and 
universities to industry problems, and to learn about innovative technologies being 
developed by them) 

• Encourage university professors to take sabbaticals in industry (visiting scientists) 

• Support summer hiring programs and industrial internships for university students 

Finally, academia will continue to take a major responsibility for the required basic 
research and theoretical studies, but should also work on industry-relevant problems. In 
that light, several participants pointed out some of the useful steps already taken by 
several universities: 

• Encourage professors to take sabbaticals in industry 

• Support internship programs to encourage students to work in industry 

• Develop flexible working relationships with industry 

• Educate industry and government on emerging research themes, accomplishments, and 
findings 

• Set up core curriculum for engineering science and arts that includes neural networks 

Section HI. Panel Summary 

The workshop ended with an assessment panel composed of sponsors from a variety of 
government agencies as well as leaders in both small and large industry. Each of the nine 
panelists gave their perspective of the neural network field. Although some agencies 
described budget constraints, a cautious optimism came through. Neural networks are a 
viable new emerging technology. After going through a period of hype followed by 
hibernation, the current atmosphere is lean and mean, which may just provide the required 
focus of energy to transfer neural networks from the world of theory to that of practice. 
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